A known image recovery process recovers an image free from any degradation when an image captured by an image capturing apparatus, such as a digital still camera, suffers degradation caused by, for example, aberrations. As an image recovery algorithm, a method is known that expresses image degradation characteristics by a point spread function (PSF) and recovers an image free from any degradation based on the PSF.
Japanese Patent Laid-Open No. 62-127976 discloses an invention that corrects blur by a filtering process having the inverse characteristics of the PSF. Also, Japanese Patent Laid-Open No. 2004-205802 discloses an invention that generates a Wiener filter from the PSF, and recovers a degraded image using the Wiener filter. Furthermore, Japanese Patent Laid-Open No. 2000-020691 discloses an invention that obtains a high-quality recovered image using characteristic information of an image capturing apparatus.
[Principle of Image Recovery]
Let (x, y) be position coordinates on a frame, let o(x, y) be an image free from any degradation (to be referred to as a subject image hereinafter), let z(x, y) be an image which is degraded due to an out-of-focus condition, aberrations, camera shaking, and so forth (to be referred to as a degraded image hereinafter), and let p(x, y) be information of a PSF of a point spread due to blur. These three pieces of information satisfy:z(x,y)=o(x,y)*p(x,y)  (1)
In the equation (1), the symbol “*” represents a convolution operation. Therefore, equation (1) can be rewritten as an integral formula expressed by:z(x,y)=∫∫o(x,y)p(x−x′,y−y′)dx′dy′  (2)
The Fourier transform of equation (2) onto a spatial frequency (u, v) domain is computed as:Z(u,v)=O(u,v)·P(u,v)  (3)where
Z(u, v) is the spectrum of z(x, y),
O(u, v) is the spectrum of o(x, y), and
P(u, v) is the spectrum of p(x, y).
Note that P(u, v) is a modulation transfer function (MTF) as the absolute value of an optical transfer function (OTF) as the two-dimensional Fourier transform of the PSF.
If p(x, y) as the PSF can be detected by an arbitrary method in addition to the degraded image z(x, y), the spectrum O(u, v) of the subject image can be calculated by calculating their spectra and using equation (4) obtained by modifying equation (3). Then, by computing the inverse Fourier transform of the spectrum calculated by equation (4), the subject image o(x, y) can be obtained.O(u,v)=Z(u,v)/P(u,v)  (4)
Note that 1/P(u, v) is called an inverse filter.
The MTF of the degradation often includes a frequency where its value becomes zero. The zero MTF value means there exists a frequency component that is not transmitted (information is lost) by degradation. If a frequency where the MTF value becomes zero exists, the subject image cannot be perfectly recovered. Therefore, the inverse filter of the MTF often includes a frequency at which a filter coefficient becomes infinity, and the spectrum value of the subject image becomes indefinite at that frequency.
In order to prevent an inverse filter coefficient from becoming infinity, image recovery often uses a Wiener filter expressed by:P(u,v)/|{P(u,v)|2+c  (5)where c is a constant having a very small value.
In order to recover the subject image from the degraded image, acquisition of an accurate PSF (or OTF, MTF) is desired.
As is well known, the PSF changes, depending on the image height, the zoom, the stop, and the subject position. Therefore, a method of calculating the PSF according to these pieces of imaging information and feeding it back to the recovery process has been proposed. For example, Japanese Patent Laid-Open No. 4-088765 estimates the PSF according to the subject distance, and uses it in the recovery of image degradation. Japanese Patent Laid-Open No. 2000-020691 executes a recovery process by correcting the PSF at the time of use of a flash by focusing attention on the fact that a luminance change of a subject during a shutter-open period is large at the time of use of the flash and is different from the PSF during the shutter-open period when the flash is not used (luminance change is uniform).
Recovery process methods are disclosed in various references. However, in practice, these references have never discussed what holds data of a recovery filter, what kind of information is to be held, how to hold the data, and how to create a recovery filter using it. In particular, there is no discussion which assumes a single-lens reflex camera using interchangeable imaging lenses and considers a plurality of combinations of imaging lenses and a camera body.
As a simplest method, recovery-process information (recovery filter coefficients, PSF data of the overall image capturing apparatus, etc.) of an overall image capturing apparatus including imaging lenses and a camera body is stored as a database in a camera body or image process software. Then, upon execution of a recovery process, recovery-process information according to imaging conditions need only be acquired from the database.
The aforementioned method is effective for a digital camera having a fixed combination of an imaging lens and a camera body. However, in case of a single-lens reflex camera using interchangeable imaging lenses, pieces of recovery-process information have to be held in correspondence with all combinations of the imaging lenses and the camera body. In this case, the amount of data becomes very large, and it is difficult for each imaging lens or camera body having a limited memory size to hold the recovery-process information. The recovery-process information is fixed data corresponding to a combination of a certain imaging lens and the camera body. For this reason, every time a new model of a camera body or imaging lens appears, recovery-process information corresponding to a combination of the new model and the existing model has to be created, and the new recovery-process information has to be reflected in the existing database. Such an operation forces the users of camera bodies and imaging lenses perform cumbersome operations.